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Abstract 

In  this  paper,  we  derive  a  general  Hajek-Renyi  type  inequality  for  vector- valued 
martingales.  Several  well  known  inequalities  are  shown  to  be  special  cases  of  this 
general  inequality.  We  also  derive  a  similar  inequality  for  dependent  sequences. 
We  then  apply  the  inequality  to  the  problem  of  strong  consistency  of  least  squares 
estimators  for  multiple  regressions. 
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1      Introduction 

For  a  sequence  of  independent  random  variables  j:;,}^!  with  zero  mean  and  finite  vari- 
ance, Hajek  and  Renyi  (1955)  proved  the  following  inequality:  for  any  e  >  0  and  for  any 
positive  integers  n  and  N  (n  <  N), 


where  Ck  (k  =  1.2, ...)  is  a  sequence  of  non-increasing  and  positive  numbers  and  Ezf  = 
erf.  The  Hajek-Renyi  inequality  is  very  useful  in  establishing  strong  laws  of  large 
numbers.  More  interestingly,  this  type  of  inequality  plays  an  important  role  in  the 
change-point  problem  for  both  parametric  and  nonparametric  estimations  [see.  e.g., 
Bhattacharya  (1987),  Dumbgen  (1991),  and  Bai  (1994,1995)]. 

This  inequality  has  been  extended  in  the  literature  into  various  settings;  see,  e.g., 
Chow  (1960)  for  martingales,  Birnbaum  and  Marshall  (1961)  for  continuous  time  mar- 
tingales, and  Sen  (1971)  for  vector- valued  random  sequences.  More  recently,  Christofides 
and  Serfiing  (1990)  extend  the  inequality  to  random  fields.  In  this  note  we  derive  a  gen- 
eral Hajek-Renyi  type  inequality  for  vector-valued  martingales.  Several  special  cases  of 
this  general  inequality  are  then  discussed.  One  of  them  concerns  a  sequence  of  quadratic 
forms  of  martingales  constructed  using  a  decreasing  sequence  of  positive-semidefinite 
matrices  (Corollary  1),  as  opposed  to  a  decreasing  sequence  of  scalars. 

This  quadratic  form  inequality  is  used  to  prove  the  strong  consistency  of  least  squares 
estimators  for  multiple  regressions.  This  latter  problem  has  received  considerable  atten- 
tion in  the  literature,  e.g.,  Anderson  and  Taylor  (1976),  Chen,  Lai  and  Wei  (1981), 
Drygas  (1976),  Lai,  Robbins,  and  Wei  (1978,1979)  and  Solo  (1981).  Using  our  inequal- 
ity, we  give  an  alternative  and  simpler  proof  of  strong  consistency. 

Furthermore,  we  derive  a  similar  quadratic  form  inequality  for  dependent  sequences. 
In  particular,  we  allow  the  dependence  sequence  to  be  a  mixingale,  which  includes  linear 
processes  and  strong  mixing  sequences  as  special  cases.  This  inequality  is  useful  in 
deriving  the  rate  of  convergence  of  change-point  estimators  for  multiple  regression  models 


with  dependent  disturbances. 

2      The  Inequality 

Let  (Q,Jr,  P)  be  a  probability  space  and  Ti  be  an  increasing  sequence  of  cr-fields  such 
that  Ti  C  T.  Let  Zi,Z2,...  be  a  sequence  of  p-dimensional  martingale  differences  relative 
to  {J-{]  such  that 

E{Zn\Tn-,)  =  0       a.s     and  E{ZiZ{)  =  S,-. 

Let  5^  =  E^=i  Z,-  Let  /it(^)  be  a  sequence  of  real-valued  convex  functions  defined 
on  Rv  such  that  h\[x)  >  /^(z)  >  •••  >  0  for  all  x  £  /?p.  Thus  /i^  is  nonnegative  and 
nonincreasing  with  respect  to  k.  The  general  inequality  is  given  in  the  following  theorem: 

Theorem  1  Let  hk(x)  be  a  sequence  of  convex,  nonnegative,  and  nonincreasing  (with 
respect  to  k)  functions  defined  on  Rp  with  Ehi(Si)  <  oo  (i  >  1),  and  let  Z^  be  a  sequence 
of  Rp -valued  martingale  differences  relative  to  an  increasing  sequence  of  a -fields  {Tic}- 
Then  for  any  e  >  0  and  for  any  positive  integers  n  and  N  (n  <  Ar),  we  have 

(1)  P  (miK  hk(Sk)  >  e)  <  l-  (E[hn(Sn)]  +   Yl   E[hi(Si)  -  hiiSi-J]]  . 

-    -  \  i=n+l  J 

When  p  =  1  (the  one  dimensional  case),  if  we  let  h(x)  =  x2,  we  obtain  the  Kolmogorov 
type  inequality.  Let  ck  be  a  sequence  of  nonincreasing  and  positive  numbers.  If  we 
choose  h(x)  =  c2kx2,  then  (1)  specializes  to  the  inequality  of  Hajek  and  Renyi  (1955) 
(for  independent  random  variables).  Further,  let  hk(x)  =  CfcX+  =  c,tmax{0,x},  then 
Chow's  inequality  is  obtained.  For  p  >  1,  let  hk{x)  =  Ck  supA^0(A'/lA)_1/'2|A'x|,  where  A 
is  positive-definite.  Then  hk(x)  is  convex  and  nonincreasing  with  respect  to  k.  For  this 
sequence  of  /i^,  (1)  yields  the  inequality  of  Sen  (1971). 

We  now  consider  another  special  case  of  the  theorem,  which  is  useful  in  establishing 
the  strong  consistency  of  least  squares  estimators.  Let  Mk  (k  =  1,2,...)  be  a  sequence 
of  positive-semidefinite  matrices  (p  x  p)  and  nonincreasing  in  the  sense  that  Mk  —  Mk+\ 
is  positive-semidefinite.  Let 

(2)  hk(x)  =  x'Mkx. 


Clearly,  hk(x)  is  convex,  nonnegative,  and  nonincreasing  with  respect  to  k.  We  have 

Corollary  1   Let  Mk   be  a  sequence   of  p  x  p  positive-semidefinite   and  nonincreasing 
matrices,  and  let  Zk  be  given  in  Theorem  1,  then 

P  (  max  S'kMkSk  >  /)  <  \  ltr(MnJ2^)  +    £    HM&i 


N 

I 

t=l  t  =  n+l 

where  tr(-)  is  the  trace  function. 


Proof  of  Corollary  1:  We  only  need  to  compute  the  right  hand  side  of  (1)  for  this 
particular  hk.  Now  E[hn(Sn)]  =  E{S'nMnSn)  =  tr{MnE[SnS'n})  =  tr{Mn  £?=1  £,-)  and 

E[hk(Sk)  -  hk(Sk-x)}  =  E[S'kMkSk  -  S'^MkSk-^} 

=  E(Z'kMkZk)-2E[S'k_1MkZk} 

=  tr(MkE[ZkZk})  =  tr(MkXk). 

We  have  used  the  fact  that  E[S'k_xMkZk]  =  E[Sk-\MkE{Zk\Tk-l))  =  0.  This  proves  the 
corollary.         □ 

Remark  1.  Because  hk(x)  =  ck  supA^0(A'y4A)_1,/2|A'2;|  =  ck{x' A~lx)ll2,  where  A  is 
positive-definite,  we  see  that  Corollary  1  extends  the  inequality  of  Sen  (1971).  The 
nonincreasingness  of  Sen's  hk  (with  respect  to  k)  is  due  to  the  nonincreasingness  of 
the  scalar  sequence  ck,  whereas  the  nonincreasingness  of  hk  defined  in  (2)  is  due  to  the 
nonincreasingness  of  the  matrix  sequence  Mk-  It  is  this  latter  feature  that  makes  the 
result  of  Corollary  1  suitable  for  the  least  squares  problem  mentioned  in  the  Introduction. 
□ 

Proof  of  Theorem  1:  The  proof  is  similar  to  that  of  Chow  (1960).  Let  A  be  the 
event  {supn<k<N  hk(Sk)  >  t).  Define  Ak  =  {hT(ST)  <  t  for  n  <  r  <  k;hk{Sk)  >  e}. 
Then  the  Ak  are  disjoint,  Ak  G  Tk  and  U^=nAk  =  A.  Thus 

N  N 


eP(A)  =  eJ2P(Ak)<^2  f    hk(Sk)dP 

k=n  fc=n  JA* 

<  Ehn(sn)  -  /     hn(sn)dp  +  f     hn+1(sn+1)dp  +  jrf  hk(sk)dp 


<  Ehn(Sn)  +  I       [hn+1(Sn+1)  -  hn+1(Sn))dP 

JCl-An 

r  N    r 

-  /  hn+1{Sn+1)dP+Yl  /    h(Sk)dP 

Jn-(AnUAn+1)  n+2'7'4* 

The  last  inequality  follows  from  hn(Sn)  >  /i„+i(Sn).  A  key  observation  is  that  for  each 
fixed  k,  the  sequence  £,  =  hk(Si)  (i  =  1,2,...)  is  a  real-valued  submartingale  relative  to 
{.F,-}  because  hk(-)  is  convex  and  Si  is  a  martingale  [Doob  (1953),  p. 295].  Thus 

(3)  /   [hn+i(Sn+1)-hn+1(Sn)]dP>0 

J  An 

and  more  generally 

(4)  /  [hk+1(Sk+1)-hk+1(Sk)]dP>0       (k>n). 

JAnU---uAk 

It  follows  that 

eP(A)  <  Ehn(Sn)  +  E{hn+1(Sn+1)  -  hn+1(Sn)} 

+  [hn+2{Sn+2)  -  hn+2{Sn+i)]dP 

jQ-{A„uAn+i) 

r  N    r 

-  /  hn+2(sn+2)  +  J2  /  hk(sk)dp. 

Jn-(AnuAn+iUAn+2)  n+3^^* 

The  theorem  follows  by  repeating  the  above  argument  and  making  use  of  (4).         □. 

Next,  we  generalize  the  inequality  to  dependent  sequences.  Let  {Jr,}^_00  be  a  se- 
quence of  increasing  cr-fields.  Assume  that  {Z,,  F,}  forms  an  Z.2-mixingale  sequence 
(Hall  and  Heyde  1980,  p.  21).  That  is,  there  exist  nonnegative  constants  {6,  :  i  >  1} 
and  {%l)m  :  m  >  0}  such  that  t/>m  1  0  as  m  — >  oo  and  for  all  i  >  0  and  m  >  0,  and 

(i)  E\\E(Zx\^.m)f  <  bUt, 

(ii)JB||Z,-JB(Z,|^+m)||2<6t2^+1, 
where  ||i||  =  (Y%=\  X1Y^2  f°r  x  £  Rp-  We  assume  in  addition  that 

(iii)  Em=i  rn1+8tpm  <  oo,  for  some .6  >  0. 

Throughout  the  remainder  of  this  paper,  any  L2  mixingale  {Zt,Ti}  is  implicitly 
assumed  to  satisfy  conditions  (i)-(iii).    The  sequence  6,  is  called  mixing  norms  and  ^>,- 


is  called  mixing  coefficient.  Mixingales  include  martingale  differences,  linear  processes, 
and  strong  mixing  processes  as  special  cases.  Many  other  dependent  structures  are  also 
special  cases  of  mixingales;  see  Andrews  (1988).  For  example,  let  Z„  =  E^-oo  ^jVn-j 
with  J2JL-oo  d]  <  °°  an<^  Vj  i-i-d.-  mean  zero  and  finite  variance  a2.  Let  Tn  —  o- 
field{?7j;  j  <  n).  Then  {Z,,  Ti]  is  a  mixingale  with  b2  —  a2  (for  all  i)  and  ij>f  =  E|i|>»  d\- 

Theorem  2  Let  Mk  be  a  sequence  of  p  x  p  non-random,  positive-semidefinite  and  non- 
increasing  matrices,  and  let  Zk  be  an  L2  mixingale.   Then,  for  every  e  >  0, 

(5)  P  (  m^  S'kMkSk  >  e2)  <  £  f  tr(MB)  £  62  +    £   62  tr(JU-)  J  • 

uAere  C  =  4(^02  +  2E~1  j2+2V2)(l  +  2E°°=1  J-2-26)  <  oo. 

Proof  of  Theorem  2:  We  can  write  (Hall  and  Heyde  1981,  p20), 

Zfc  =    E    Zik,       with  Zjk  =  E{Zk\Jrk^J)  -  E(Zk\Jrk-i-i)- 

j=  —  oo 

Thus  Ml/2Sk  =  Mfc1/2£f=1  Z,  =  E°l_oc  Affc1/2£?=1  Zji  =  ££_«,  Mfc1/25,fc,  where  Sjk  = 


Ef=1Z„.  Thus 


P  (  max  ||Mfc/25fc||  >  e)  <  P      V     max   ||Mfc1/2S,fc||  >  t \  . 

\n<k<N"      *  m  —    y  —        i     z_^  „<fc<7v  "      *        J    m  -      i 

Note  that  for  each  j,  {5"^,^-^;  1  <  ^  <  N]  is  a  martingale.  Let  a;  >  0  for  all  j  such 
that  E£_ooaJ  =  1.  Then 


(6)  P      E   nm«  IIM^S.fcU  >  e     <    £    P  U  K'%11  >  a,e 

\j  =  -oo      -    -  /  j  =  -oo  -    - 

1         oo  /  n  N  \ 

<T2    E   °72    *r(Mn£Ei,-)+    E   MM.-E,-.-) 

j  =  -oo  \  j=l  :=n+l  / 

where  Ej,-  =  E{ZltZ'-i)  and  the  second  inequality  follows  from  Corollary  1.  For  any  vector 
i  £  Rp,  we  have  x'xl  —  xx'  >  0  (positive-semidefinite),  where  /  is  apxp  identity  matrix. 
Substitute  Zj,  for  x  and  take  expectation  to  obtain  .E||Zj,||2  I  —  Ej,  >  0.  Furthermore,  by 


tA  t  sVS, )[tr{Mn)Y,bU  £  *MM0    ■ 


1  =  1  t=n+l 

"2,/,2  . 


the  mixingale  property  and  the  definition  of  Z;I,  it  is  easy  to  show  that  £]|Zj,-||2  <  462V>2Ji 
for  all  j.  Thus  4&?^j£|2  -  £*  >  0.  Using  the  fact  that  tr(CA)  <  tr(CB)  for  B  >  A  and 
C  >  0,  we  have  tr{MiLJX)  <  4b2tpf^tr(Me).  It  follows  that  the  right  hand  side  of  (5)  is 
bounded  by 

a      I     oo  \      /  n  A' 

72 

It  remains  to  choose  appropriate  a^s  such  that  £j  a~2ij}2-,  is  bounded.    Let  i/0  =  1  and 

Vj  =  j~l~s  (j  >  1),  where  6  >  0  as  given  in  (iii).  Let  a;  =  i/j/(l+2X3fc=i  "*)  an^  «-j  =  Qj 

for  j  >  0.  Then  £j  a,  =  1.  By  assumption  (iii),  Ej  ajVjj|  =  (V»o  +2E,"i  J2+2^2)(l  + 

2EJc=iJ-2-2')<oo-  D 

Remark  2.    Inspecting  the  proof  we  see  that  if  £,,   <  S,-^»jl-|,  for  some  positive- 

semidefinite  E,,  then  from  tr(MtEji)  <  tr(MeT,i)tp?j,  and  (6),  we  have 

(7)  P  (  max  S'kMkSk  >  e2)  <  %  (tr{Mn  £  £,-)  +    f)   tr(M,-S.-)  )  . 

\n<fc<W  /  £      y  .=1  1=n+l  / 

which  is  analogous  to  Corollary  1.  This  situation  arises  in  the  following  setup:  Z,  =  x,£,-, 
where  i,  is  a  p  x  1  vector  of  nonrandom  constants  and  e,  is  random  variable  such  that 
{ez,Jrt}  forms  an  L2  mixingale  with  mixing  norms  {b2}  and  mixing  coefficients  {V'™}- 
We  can  write  e,  =  Hj=-oo  ej<i  so  *nat  %i  =  XT^-co  ^i«  with  Z;1  =  i,£j,.  It  follows  that 
£*  =  EiZjiZ'ji)  =  Xix'iE^A  <  (xix'^bfyfa.  We  can  choose  E,  =  (46?)x,-x;.  This 
setup  is  related  to  regression  models,  where  x,  are  regressors  and  £,  are  disturbances. 
Inequality  (7)  will  be  used  in  proving  strong  consistency  of  least  squares  estimators  in 
regression  models  with  mixingale  disturbances.  □ 

Corollary  2  For  Z,  in  Theorem  2,  assume  that  the  mixing  norms  b2  are  such  that 
sup,  b2  <  b  <  oo.   Then,  for  some  M  <  oo, 

M 


'UsM'i?*1-')- 


for  all  N  >  n  >  1. 


Proof:  Let  Mk  =  k~2I.  Then  tr(Mn)  £?=!  62  <  p62/n  and  ££B+1  &2*r(M)  < 
pESn+i  &2/1'2  =  0(l/nj.  Thus  the  right  hand  side  of  (5)  is  bounded  by  M/(e2n)  for 
some  M  <  oo.        □. 


The  corollary  is  useful  in  change-point  problems.  Note  that  since  the  probability 
bound  does  not  depend  on  N.  the  corollary  holds  when  we  replace  Ar  by  oo.  This  yields 
the  strong  law  of  large  numbers  for  mixingales  as  well  as  certain  rate  of  convergence  for 
the  strong  law. 

3      Application 

The  application  concerns  the  strong  consistency  of  least  squares  estimator  of  the  param- 
eter Bj  in  the  following  multiple  regression: 

(8)  yt  =  XflA  +  •  •  •  +  xipBP  +  £i  =  xtf  +  £{     (z  =  1,2,...) 

where  the  vector  x,-  =  (xtl,  ...,xtp)'  (i  >  1)  consists  of  known  constants,  B  =  (/5l5 ...,  Bp)' 
is  the  regression  coefficient,  and  {e,}  is  an  L2  mixingale  with  respect  to  an  increasing 
sequence  of  cr-fields  {Ti}.  Denote  by  Xn  =  (xi,  ...,xn)'  the  design  matrix  and  let  Y„  = 
(j/i, ...,  yn)'  and  Un  =  (e\, .. .,£„)'.  Assume  (X'nXn)  is  positive-definite  for  some  n  =  m 
and  hence  for  all  n  >  m.  The  least  squares  estimator  bn,  based  on  the  first  n  observations, 
is  given  by 

bn  =  (X'nXn)-lX'nYn  =  6+  {X'nXn)-'X'nUn. 

Thus  strong  consistency  is  equivalent  to 

(9)  {X'nXn)-lX'nUn  -»  0     with  probability  1. 

When  the  regressors  are  non-random,  Lai,  Robbins,  and  Wei  (1977,  1979)  establish  the 
strong  consistency  under  the  minimal  condition 

(io)  (x'nxnyl  -  o 

together  with  the  assumption  that  £,  is  a  martingale-difference  sequence  with  respect  to 
an  increasing  sequence  of  cr-fields  {.T7,}  such  that  sup,  E{e2 |^i-i)  <  °°  (a.s.).  Condition 

(10)  is  significantly  weaker  than  the  conventional  assumption  that  (.X'nXnjn)  converges 
to  a  positive  definite  matrix.    Under  the  latter  assumption,  the  condition  number  of 


(X'  Xn)  (the  ratio  of  the  largest  and  smallest  eigenvalues)  will  be  uniformly  bounded, 
and  the  strong  consistency  will  be  trivial  to  prove.  Condition  (9)  is  known  as  a  necessary 
and  sufficient  condition  when  the  e,-  are  i.i.d.  with  zero  mean  and  finite  variance;  see  Lai 
et  al.  (1979).  For  dynamic  models,  strong  consistency  is  also  studied  by  many  authors 
(cf.  Anderson  and  Taylor,  1979,  Chri  Christopeit  and  Helmes,  1980,  Lai  and  Wei,  1982). 
Using  our  inequality,  we  give  an  alternative  and  simpler  proof  of  strong  consistency. 
We  do  impose  a  stronger  condition  than  (9),  but  we  allow  the  disturbances  to  have  broad 
correlation  structures.  Let  Amax(n)  denote  the  largest  eigenvalue  of  X'nXn  and  A^^n) 
the  minimum  eigenvalue  of  X'nXn.  Let  $  be  the  set  of  functions  <f>(x)  such  that  each 
<j)(x)  is  positive  and  non-decreasing  in  the  interval  x  >  x0  for  some  x0  and  the  series 
Y^  — r^rr  converges  for  every  a  >  0.  For  example  <f>(x)  =  x5,  <f>(x)  =  [\og(x)]1+6  and 
(f>  —  log(x)[loglog(x)]1+l5  (S  >  0)  are  members  of  $.  We  assume,  for  some  </>£$, 

(11)  Amjn(n)  ->  oo        and   <f>(Xmax(n))  =  0(Amin(n)). 

This  condition  is  weaker  than  (2.24)  of  Lai  and  Wei  (1982)  but  slightly  stronger  than 
their  (1.17).  For  dynamic  models,  Lai  and  Wei  show  that  condition  (11)  is  close  to  be 
weakest  possible  for  strong  consistency. 

Next,  we  assume  {e,}  is  an  L2  mixingale  with  respect  to  an  increasing  sequence  of 
(j-fields  {J~i}-  Further  assume  that  sup,  b2  <  b2  <  oo.  When  £,-  is  a  martingale-difference 
sequence,  b2  can  be  taken  as  Ee2.  Note  that  sup,  Ee2  <  oo  can  be  significantly  weaker 
than  sup,-  E(e2\J-,-i)  <  oo  (a.s.),  as  is  assumed  in  the  previous  literature.  For  example, 
let  £,  =  rjini-i,  where  77,  are  i.i.d.  normal.  Let  J-x  =  (T-fieldj^;  j  <  i) .  Then  e,  is  a 
sequence  of  martingale  differences.  The  condition  that  sup,  E(e2\!F,-i)  <  00  is  equivalent 
to  sup,  T]2  <  00,  which  does  not  hold. 

Theorem  3  Let  {ei,J-i}  be  an  L2  mixingale  with  sup,  b2  <  b2  <  00.  Assume  (11)  holds. 
Then  the  least  squares  estimator  bn  is  strongly  consistent  for  /?. 

Proof.  We  observe  that 

\\bk-0\\2    -    \\{X'kXkY'X'kUk\\2 


(12)  <    \\(X'kXk)-^f\\{X'kXk)-^XkUk\\2 

=      (Xninm-'iU'MX'M-'XiUk) 

Let  Zi  =  x,e,.  Then  Z,  is  a  i?p-valued  mixingale  with  respect  to  {Tx}.  Let  Sk  = 
H!=i  xi'£t  =  X'kPk,  and  let  Mfc  =  ^(XlXk)'1  with  c*  =  Amin(A;).  We  see  that  Mk  is 
non-increasing,  and  ||6^  —  8\\2  <  S'kMkSk  by  (12).  Thus  by  Remark  2, 


(13)  P(    sup    \\bk  -/3||2>e2)     <     P(    sup    S[M,S,  >  e2) 

<     -2(b2tr(MnJ2^)+b2    £    *r(M,-£,)) 

£       \  1=1  i=n+l  / 

where  E,  =  x,x^.  Note  that 

(14)  tr(Mn  £  Si)  =  c^1ir((X:Xn)-1^Xn)  =  pc^1  =  P/Amn(n). 

t=i 

Furthermore, 

ir(M,£:)  =  x'iMiXi  =  c-'x^XlX^x,  =  c-l(\At\  -  \A,^\)/\A,\ 

where  Ax  =  {X[X{)  and  |j4,-|  is  the  determinant  of  A,.  Because  \An\  <  Amax(n)p,  or 
equivalently,  l^l1^  <  Amax(n),  we  have  <K|/ln|1/p)  <  <£(Amax(n))  =  0(Amjn(n))  by  (11). 
This  implies  that 

c-1(|A,-|-|A_1|)/[A,|<l- 


.-.,,    1    ,         ,    ,         ,w,    ,    ^  r(l^,|-|^-l| 


\Ai\<f>(\Ai\Vr) 
for  some  L  <  oo.  Thus 


00      (|4  I  —  I  4     , 

(15)        £  (r(M-E,)=  £  ^'(KI-IA-iD/IAISI  £       " 


Vr'(l'4'l"K-,i)/l'4'1-iS:+,M.wi-y"»)- 

Because  J2  l/(n</)(nl/'p))  <  oci  the  above  series  converges  by  the  integral  comparison 
test.  Combining  (13)-  (15),  and  letting  N  ■— ►  oo,  we  have 

p(         III         «!!>-,     A<rC/_^!_    i-tff     V-     (Mi|-|A;-i|)' 

The  right  hand  side  above  converges  to  zero  as  n  — >  oo.         □ 
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